Components of machine learning systems are not (yet) perceived as security hotspots. Secure coding practices, such as ensuring that no execution paths depend on confidential inputs, have not yet been adopted by ML developers. We initiate the study of code security of ML systems by investigating how nucleus sampling---a popular approach for generating text, used for applications such as auto-completion---unwittingly leaks texts typed by users. Our main result is that the series of nucleus sizes for many natural English word sequences is a unique fingerprint. We then show how an attacker can infer typed text by measuring these fingerprints via a suitable side channel (e.g., cache access times), explain how this attack could help de-anonymize anonymous texts, and discuss defenses.

Correctness: A few assumptions in this paper are unlikely to hold in real life: 1. Being aware of the _exact_ model that was used to produce the given text. Does this hold for models that have been finetuned? Does it hold for identical models trained on the same data with two different random seeds? Does it hold for models that are merely similar in size and data? - These things take significantly more resources to investigate, but at least looking at finetuning which is quick and built-in to most libraries (including HuggingFace, which the authors use) would be key to making these results actually realistic. That said, currently people _do_ tend to use exactly the same model in many cases, i.e.

This paper generated a significant amount of discussion. SCIENTIFIC: Regarding the purely scientific aspects, the reviewers discussed about the novelty of the contribution. On the one hand, if one takes the point of view of the security community, the proposed attack and defense are known and the vulnerability is not surprising since any data-dependent accesses is prone to side-channel attacks. On the other hand, from the point of view of the machine learning community where these concerns are currently not well known, the paper presents very clearly a reasonable approach to start thinking about security of machine learning and NLP code using actual algorithms that text generation researchers and practitioners use. The paper can thus serve a useful cross-discipline discussion.

Components of machine learning systems are not (yet) perceived as security hotspots. Secure coding practices, such as ensuring that no execution paths depend on confidential inputs, have not yet been adopted by ML developers. We initiate the study of code security of ML systems by investigating how nucleus sampling---a popular approach for generating text, used for applications such as auto-completion---unwittingly leaks texts typed by users. Our main result is that the series of nucleus sizes for many natural English word sequences is a unique fingerprint. We then show how an attacker can infer typed text by measuring these fingerprints via a suitable side channel (e.g., cache access times), explain how this attack could help de-anonymize anonymous texts, and discuss defenses.